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1  Statement  of  Objectives 

1.  Port  the  parallel  wave  optics  adaptive  optics  (AO)  simulation  code  developed  at 
Gemini  Observatory  to  the  Huinalu  Linux  cluster  at  MHPCC.  The  code  consists  of 
simulation  routines  written  in  Matlab,  MPI  message-passing  routines  written  in  C  for 
communication  between  the  simulation  processes,  and  makefiles  to  simplify  code 
compilation  (All  but  a  single  manager,  or  master,  process  is  implemented  as  compiled 
Matlab.  This  allows  the  simulation  to  be  used  with  a  single  Matlab  license  on  an 
arbitrary  number  of  nodes). 

2.  Validate  the  installation  of  the  code  on  the  Huinalu  cluster  by  generating  test  case 
results  for  sample  AO  configurations  based  upon  the  first-order  parameters  for  the 
AEOS  AO  system,  as  well  as  the  current  design  for  the  Gemini-South  multi¬ 
conjugate  AO  (MCAO)  system. 

3.  Provide  detailed  descriptions  of  simulation  input  parameters  and  output  formats  to  the 
staff  at  MHPCC.  Assist  the  MHPCC  staff  in  developing  user  documentation  and 
user-friendly  code  interfaces. 

4.  Upgrade  the  simulation  code  to  model  Shack-Hartmann  wavefront  sensing  using  an 
artificial,  sunlit  satellite  as  the  wavefront  sensing  beacon,  beginning  with  the  current 
model  for  wavefront  sensing  using  extended  laser  guide  stars. 

5.  Upgrade  the  simulation  code  to  model  windshake  and  similar  tip/tilt/focus 
disturbances  in  3  to  10  meter  class  telescopes  (but  not  future  giant  telescopes,  where 
more  sophisticated  models  of  telescope  structural  dynamics  may  be  necessary.) 

6.  Support  the  MHPCC  staff  in  implementing  special  models  for  specific  features  of  the 
AEOS  AO  system.  Examples  might  include  the  tip/tilt  sensor  and  tracking  algorithm, 
and  adaptive  background  subtraction  algorithms  for  wavefront  sensing  during 
twilight. 

7.  Upgrade  the  simulation  code  to  model  correlation  algorithms  for  computing  Shack- 
Hartmann  tip/tilt  measurements  when  using  an  extended  object,  such  as  an  artificial 
satellite  or  an  elongated  sodium  laser  guide  star,  as  the  wavefront  sensing  beacon. 

8.  Upgrade  the  simulation  code  to  generate  AO-compensated  images  of  extended  scenes 
such  as  galaxies  and  artificial  satellites,  and  provide  more  extensive  evaluation  of  AO 
and  MCAO  system  performance  across  extended  fields-of-view. 

9.  Upgrade  the  simulation  code  to  provide  more  extensive  evaluation  of  the  short-  and 
long-exposure  performance  of  AO  and  Extreme  AO  (ExAO)  for  narrow  field 
imaging.  Upgrade  the  code  to  model  a  coronagraph  or  similar  astronomical 
instrument  for  detecting  faint  stellar  companions  and  disks. 

10.  Implement  computationally  efficient  wavefront  reconstruction  algorithms  for  very 
high  order  (10,000  to  100,000  deformable  mirror  actuators)  MCAO  and  ExAO 
systems  on  future  giant  telescopes  (FGTs)  as  these  algorithms  are  developed  under 
ongoing  projects  funded  by  NSF  and  AFOSR. 

11.  Reduce  simulation  execution  times  by  using  a  parallel  fast  Fourier  transform  (FFT) 
routine  for  wavefront  propagation  and  phase  screen  generation,  and  upgrading  (as 
necessary)  the  scheduling  algorithm  that  allocates  tasks  to  worker  processes. 

12.  Simplify,  organize,  and  document  the  simulation  code  as  time  permits. 
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2  Status 

This  is  the  second  and  final  status  report  for  a  project  to  port  an  existing  parallel  adaptive 
optics  (AO)  simulation  code  to  the  Huinalu  cluster  at  the  Maui  High  Performance 
Computing  Center,  and  to  upgrade  the  simulation  to  provide  an  improved  modeling 
capability  for  both  the  AEOS  AO  system  and  the  astronomical  AO  systems  proposed  for 
future  extremely  large  telescopes  (ELTs).  The  functional  elements  of  the  simulation 
include  the  basic  AO  components,  disturbances,  and  control  algorithms  necessary  for 
scalar  wave  diffraction  modeling  of  ground-based  natural  guide  star  (NGS)  AO,  laser 
guide  star  (LGS)  AO,  and  multi-conjugate  AO  (MCAO)  imaging  systems,  as  illustrated 
in  figure  1  below. 

As  described  in  the  original  proposal,  the  Huinalu  cluster  is  an  attractive  platform  for  this 
simulation  for  several  reasons: 

•  The  large  number  of  processing  nodes  enables  effective  coarse-grained 
parallelization  of  wavefront  propagation  operations  for  the  multiplicity  of  guide 
stars  and  science  targets  observed  by  MCAO  and  other  wide-field  AO  systems; 

•  After  planned  upgrades  to  the  simulation  code,  Huinalu’s  multiple  processors  will 
also  be  used  for  fine-grained  parallelization  of  wavefront  reconstmction 
algorithms  in  simulations  of  proposed  ELT  AO  systems  with  between  104  to  105 
deformable  mirror  (DM)  degrees  of  freedom; 

•  Finally,  the  software  port  of  the  simulation  to  Huinalu  has  been  fairly 
straightforward,  since  it  is  written  in  MATLAB  and  relies  on  the  MPI  1 .2  protocol 
for  interprocess  communication. 

During  FY  2004  we  have  focused  on  the  following  tasks: 

Initial  port  to  Huinalu  (Task  l--completed):  We  established  user  accounts  at 
MHPCC,  transferred  the  original  version  of  the  simulation  source  code  to  Huinalu, 
and  compiled  the  code  using  the  Huinalu  Matlab  compiler  and  libraries. 

Validate  the  installation  (Task  2— completed):  We  confirmed  that  the  code 
generated  correct  results  for  test  case  simulations  of  8-meter  class  natural  guide  star 
(NGS)  AO  and  MCAO  systems.  We  also  used  these  test  cases  to  quantify  execution 
times  vs.  the  number  of  Huinalu  nodes  N  used  by  the  simulation,  and  confirmed  that 
execution  time  is  proportional  to  N~]  provided  that  N  is  no  larger  than  the  number  of 
wavefronts  traced  through  the  atmosphere  from  separate  guide  stars  and  evaluation 
directions  each  simulation  cycle.  This  value  may  be  20  to  50  for  simulations  of 
wide-field  MCAO  systems. 

Describe  the  code  to  MHPCC  staff  (Task  3):  Much  of  the  validation  and  timing 
work  described  above  was  accomplished  in  the  company  of  observers  from  MHPCC 
and  its  on-site  contractors. 
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Figure  1:  AO  simulation  code  features 
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Computationally  efficient  wavefront  reconstruction  algorithms  (Task  10); 

Using  separate  funding,  we  continued  developing  and  testing  parallel  versions  of  the 
computationally  efficient  wavefront  reconstruction  algorithms  previously 
demonstrated  in  serial  simulations  of  AO  systems  with  up  to  8  104  degrees  of 
freedom.  During  this  period,  we  completed  tests  of  a  parallel  multigrid 
preconditioned  conjugate  gradients  (PCG)  algorithm  implemented  using  the  Parallel 
Toolkit  for  Scientific  Computation  (PETSc).  We  developed  and  tested  a  socket- 
based  interface  to  allow  this  routine  to  be  called  from  AO  simulations  written  in 
either  Matlab  or  C.  Work  is  continuing  to  implement  a  broader  class  of  control 
algorithms  and  improve  the  generality  and  flexibility  of  the  socket  interface. 

Further  details  on  tasks  2  and  10  can  be  found  in  the  following  section. 

3  Accomplishments  and  Findings 

3.1  Task  2  (Code  Validation) 

The  parallel  version  of  the  AO  simulation  has  been  validated  for  correctness  against 
serial  versions  of  the  code.  The  numerical  difference  between  the  two  sets  of  results  is 
machine  epsilon  (i.e,  roundoff  error).  The  following  table  summarizes  sample  timing 
results  for  simulations  running  up  to  64  processors  on  the  Huinalu  supercomputer: 


Number  of 

Startup  Time 

Full  Simulation 

Seconds  per 

Simulation 

Processors 

(2  cycles) 

Time  (30  cycles) 

Simulation  Cycle 

5 

63.94 

947.50 

31.56 

NGS  AO 

5  Evaluation 

9 

34.34 

502.18 

16.71 

Directions 

17 

34.63 

502.28 

16.70 

NGS  AO 

33 

209.00 

1075.40 

30.94 

49  Evaluation 

Directions 

65 

50.76 

653.16 

21.51 

MCAO 

5 

95.29 

1408.30 

46.89 

9  Evaluation 

Directions 

9 

50.26 

733.22 

24.39 

MCAO 

33 

109.45 

1556.40 

51.68 

49  Evaluation 

Directions 

65 

64.83 

879.53 

29.10 

The  first  test  case  simulation  was  a  simple  NGS  AO  case  that  required  over  one  minute 
per  simulation  cycle  to  execute  on  a  single  processor.  It  runs  as  quickly  as  17  seconds 
per  iteration  on  Huinalu  if  the  number  of  processors  is  sufficiently  large.  The  final  test 
considered  a  more  complex  MCAO  system  with  both  natural  and  laser  guidestars,  and 
and  simultaneous  performance  evaluation  at  49  points  in  a  scientific  field.  This 
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simulation  executed  in  approximately  30  seconds  per  iteration  on  Huinalu,  over  50  times 
faster  than  the  run-time  on  a  single-processor  with  similar  characteristics  performance 
characteristics. 

3.2  Task  10  (Parallel,  computationally  efficient  reconstructors) 

We  have  ported  several  computationally  efficient  wavefront  reconstruction  algorithms 
from  MATLAB  into  the  Portable  Extensible  Toolkit  for  Scientific  Computing  (PETSc),  a 
library  of  parallelized  linear  algebra  routines  written  in  C.  PETSc  gives  access  to  a  wide 
range  of  sparse  iterative  and  direct  solvers  that  may  be  used  as  wavefront  reconstruction 
algorithms.  Its  routines  may  also  be  used  as  building  blocks  for  custom  approaches,  such 
as  our  MCAO  solver  based  upon  multigrid-preconditioned  conjugate-gradients  and  a 
sparse  plus  low-rank  Gauss-Seidel  smoother.  Dr.  Curt  Vogel’s  mulitgrid  reconstructor 
for  extreme  AO  (ExAO)  has  also  been  implemented  in  PETSc. 

Initial  timing  tests  indicate  factors  of  8-10  improvement  in  execution  times  after  porting 
these  wavefront  reconstruction  algorithms  from  MATLAB  to  the  optimized  PETSc 
routines,  even  for  the  case  of  a  single  processor. 

Finally,  work  is  in  progress  to  develop  a  supporting  library  of  routines  implementing  to 
simplify  access  to  PETSc  from  AO  simulations  written  in  C  or  C++,  as  well  as  MEX  and 
socket  connections  designed  for  use  with  Matlab  AO  simulations. 

4  Personnel  Supported 

Aron  Ahmadia,  an  undergraduate  intern  with  extensive  experience  in  parallel  computing, 
worked  on  tasks  1,  2,  3,  and  10.  Brent  Ellerbroek  worked  on  tasks  2  and  3. 

5  Interactions  and  Transitions 

5.1  Meetings  and  Conferences 

We  attended  the  January  2004  PRET  review  at  MHPCC,  and  presented  a  brief  summary 
of  the  code  architecture,  sample  numerical  results,  and  code  execution  times  using  the 
Huinalu  cluster.  The  presentation  is  attached  as  reference  A. 

Aron  Ahmadia  also  met  in  an  informal  collaboration  with  the  Mathematics  department  at 
Montana  State  University  and  presented  a  discussion  on  his  progress,  development,  and 
motivation  for  COW.  The  presentation  is  attached  as  reference  B. 

5.2  Transitions 

The  PETSc  codes  developed  under  task  10  (parallel,  computationally  efficient 
reconstruction  algorithms)  code  will  be  used  by  the  Thirty  Meter  Telescope  (TMT) 
Project  to  model  the  performance  of  the  AO  and  MCAO  systems  proposed  for  future 
giant  astronomical  telescopes  such  as  TMT  (TMT  is  a  partnership  between  the 
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Association  of  Universities  for  Research  in  Astronomy  (AURA),  the  Association  of 
Canadian  Universities  for  Research  in  Astronomy  (ACURA),  the  University  of 
California,  and  the  California  Institute  of  Technology).  It  is  expected  to  be  interfaced 
with  three  or  possibly  four  AO  simulation  codes  developed  at  Gemini  Observatory,  the 
European  Southern  Observatory  (ESO),  the  California  Institute  of  Technology,  and  the 
National  Optical  Astronomical  Observatory  (NOAO). 
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